feat(DurableExecution): incremental, heterogeneous Parallel API (#2519) by GarrettBeatty · Pull Request #2553 · aws/aws-lambda-dotnet

GarrettBeatty · 2026-09-01T22:32:31Z

Description

Implements #2519: an additive, branch-oriented parallel API for Amazon.Lambda.DurableExecution supporting heterogeneous per-branch result types and incremental branch registration, alongside the existing homogeneous ParallelAsync<T> overloads (which are unchanged).

await using var parallel = ctx.CreateParallel(name: "process-order");

IParallelBranch<InventoryReservation> inventory = parallel.BranchAsync(
    "inventory", async (branch, ct) => await ReserveInventoryAsync(branch, ct));
IParallelBranch<PaymentAuthorization> payment = parallel.BranchAsync(
    "payment", async (branch, ct) => await AuthorizePaymentAsync(branch, ct));

IBatchResult summary = await parallel.CompleteAsync();

InventoryReservation reserved = await inventory;   // own concrete type — no shared base, cast, or envelope
PaymentAuthorization  authed   = await payment;

What & why

Today every branch of a Parallel must share one generic result type T, forcing unrelated branch contracts into object, a common base type, or a wrapper. This adds a branch-scoped generic API (matching the Java SDK's ParallelDurableFuture) that gives each branch its own compile-time type, replay-safe per-branch deserialization, and incremental composition (register/start branches as work is discovered, then seal).

New public API

IDurableContext.CreateParallel(name?, config?) → IDurableParallel
IDurableParallel : IAsyncDisposable — BranchAsync<T>(name, func), CompleteAsync(ct)
IParallelBranch<T> — awaitable typed handle exposing Name / Index / Status

Design

Each branch runs as an existing ChildContextOperation<T> with the same deterministic child op id (hash("{parentId}-{index}")) and the same parent CONTEXT/Parallel BatchSummary checkpoint shape as batch Parallel — so replay, checkpoints, and reconstruction are identical and interoperable.
Branches start on registration, gated by a shared MaxConcurrency semaphore and a cooperative short-circuit token; CompleteAsync seals, awaits per CompletionConfig, and checkpoints the aggregate.
Deterministic replay: branch identity is positional (register the same branches in the same order); a name change at an index throws NonDeterministicExecutionException. Terminal-parent replay reconstructs from the frozen inline summary without re-running (re-running only overflow-stripped branches).
DisposeAsync auto-completes if CompleteAsync wasn't called, so await using always writes the terminal checkpoint.
MaxConcurrency, CompletionConfig, NestingType, cancellation, and the registered ILambdaSerializer are honored unchanged.
Refactors BatchSummary (de)serialization + overflow handling out of ConcurrentOperation<T> into a shared BatchSummaryCodec so the batch and incremental paths can't diverge on the wire format.

Testing

16 unit tests (IncrementalParallelOperationTests): fresh happy path, heterogeneous types, deterministic ids, MaxConcurrency, completion short-circuit/skip, failure surfacing, empty, replay reconstruct (inline + failed branch), name-drift, and STARTED-parent replay. All 428 unit tests pass.
2 integration tests, both verified green against the durable-execution service:
- IncrementalParallelHeterogeneousTest — string/int/POCO branches round-trip end-to-end.
- IncrementalParallelReplayTest — deterministic replay across both the STARTED-parent Run path and the terminal-reconstruct resume (each branch step executes exactly once).
Docs: new "Incremental, heterogeneous branches" section in docs/core/parallel.md.

Note for reviewers

Because IParallelBranch<T> is awaitable, a bare parallel.BranchAsync(...) statement whose result is ignored trips CS4014 under the repo's warnings-as-errors — callers must capture or discard (_ =) the handle. Flagging in case the team prefers a non-awaitable handle + explicit GetResultAsync().

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copilot

🟡 Changes recommended

Replay validation, cancellation, completion concurrency, and branch result consistency contain unresolved correctness issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an incremental parallel API supporting heterogeneous typed branches while preserving existing batch APIs and checkpoint formats.

Changes:

Adds CreateParallel, typed branch handles, and orchestration logic.
Extracts shared batch-summary serialization.
Adds unit, integration, and documentation coverage.

File summaries

File	Description
`IncrementalParallelOperationTests.cs`	Tests incremental execution and replay.
`IncrementalParallelReplayFunction.csproj`	Configures replay test function.
`IncrementalParallelReplayFunction/Function.cs`	Exercises replay paths.
`IncrementalParallelHeterogeneousFunction.csproj`	Configures heterogeneous test function.
`IncrementalParallelHeterogeneousFunction/Function.cs`	Exercises typed branches.
`IncrementalParallelReplayTest.cs`	Validates replay integration.
`IncrementalParallelHeterogeneousTest.cs`	Validates heterogeneous integration.
`IParallelBranch.cs`	Defines typed awaitable handles.
`IncrementalParallelOperation.cs`	Implements incremental orchestration.
`ConcurrentOperation.cs`	Uses shared summary codec.
`BatchSummaryCodec.cs`	Centralizes summary serialization.
`IDurableParallel.cs`	Defines the public parallel API.
`IDurableContext.cs`	Exposes `CreateParallel`.
`DurableContext.cs`	Constructs incremental operations.
`docs/core/parallel.md`	Documents the new API.

Review details

Suppressed comments (2)

Libraries/src/Amazon.Lambda.DurableExecution/Internal/IncrementalParallelOperation.cs:160

Serialize the value before completing the public result task. With NestingType.Flat, Serialize(value) can throw after TrySetResult; the catch path then records a failed outcome but cannot replace the already-successful handle result, so summary.HasFailure/Status report failure while await branch returns a value.

            var value = await run().ConfigureAwait(false);
            if (_frozenStatus is null) _status = (int)BatchItemStatus.Succeeded;
            _result.TrySetResult(value);
            return BranchOutcome.Success(Index, Name, Serialize(value));

Libraries/src/Amazon.Lambda.DurableExecution/Internal/IncrementalParallelOperation.cs:331

Handle missing/corrupt terminal summaries explicitly. ParseSummary returns null for these payloads, but the operation remains in Terminal mode; every branch is then resolved as skipped and CompleteAsync synthesizes AllCompleted, masking a previously terminal checkpoint. Recover from child checkpoints where possible or fail replay rather than returning a false success.

        if (terminal)
        {
            _mode = ParallelExecutionMode.Terminal;
            _frozenSummary = BatchSummaryCodec.ParseSummary(existing!.ContextDetails?.Result);
            _startTask = Task.CompletedTask;

Files reviewed: 16/16 changed files
Comments generated: 7
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

…al-parallel overflow/await fixes, serializer deferral, Branch rename, docs DurableExecution PR #2553 (stacked on feature/per-step-serializer-conformance). 1. [BLOCKER] StepOperation.ExecuteFunc: move the fresh-success SUCCEED enqueue and result round-trip OUTSIDE the try that funnels into HandleStepFailureAsync (mirrors ChildContextOperation). A serializer that cannot deserialize its own just-written payload now surfaces the fault directly instead of enqueuing a RETRY/FAIL that conflicts with the already-committed SUCCEED. 2. [MAJOR] .autover/changes/12a4a1f7: Minor -> Major. The package is GA (1.x) per CLAUDE.md, and the unconditional fresh-success round-trip is an observable happy-path behavior change for ALL serializers on non-suspending workflows (reference identity, DateTime.Kind, [JsonIgnore], precision). Not preview-exempt. 3. [MAJOR] IncrementalParallelOperation overflow recovery: isolate overflow-recovery re-runs (frozenStatus set) from _shortCircuitCts/_dispatchCts so a completion-policy short-circuit can no longer cancel them; and exclude frozen branches from the cooperative-bail arm so _result honors _frozenStatus (never resolves a frozen Succeeded branch to SkippedError, which made `await branch` throw while Status==Succeeded and lost the recovered value). 4. [MINOR] DurableContext.CreateParallel: defer LambdaSerializerHelper.GetRequired via a lazy factory (memoized in the operation). A workflow overriding the serializer on every branch no longer requires a global serializer at CreateParallel time (AOT / per-branch scenario). GetRequired is resolved only when a branch falls back. 5. [MINOR] Rename IDurableParallel.BranchAsync<T> -> Branch<T>. The method returns a handle synchronously (not a Task), so the Async suffix was misleading. Safe: the API is new/unreleased (absent on master). Updated the interface, impl, all call sites (conformance + tests), docs (parallel.md), and the AutoVer changelog text. 6. [MINOR] IDurableParallel.Branch XML doc: document ArgumentNullException, ObjectDisposedException, and NonDeterministicExecutionException in addition to InvalidOperationException. 7. [MINOR] IncrementalParallelBranch.ExecuteAsync: fault _result before rethrowing a workflow-level DurableExecutionException, so a caller that catches the fault out of CompleteAsync and then awaits the handle observes the fault instead of hanging. 8. [MINOR] Correct the IncrementalParallelOperation class summary and IParallelBranch.Index doc to reflect the 1-based operation-ID suffix (hash("{parentId}-{index+1}")). 9. [NIT] IncrementalParallelHeterogeneousTest: replace the tautological Contains("200") (satisfied by the "USD:4200" POCO branch) with the distinguishing token "Payment":200. Tests: added 4 unit tests (fresh-success round-trip deserialize failure surfaces without RETRY/FAIL; CreateParallel with no global serializer + per-branch overrides does not throw; deferred fallback still throws on a non-overriding branch; a branch faulting with a workflow-level error faults the handle instead of hanging). Build and Amazon.Lambda.DurableExecution.Tests pass (447/447, net10.0). Integration-test and deployed-function projects compile; the heterogeneous integration test requires an AWS deployment and was not run here. AutoVer: source changes are refinements to the two features already covered by the existing change files, so both existing entries were updated (12a4a1f7 -> Major; add-incremental changelog text updated for the Branch rename) rather than adding a new change file. Reclassifying 12a4a1f7's Type was a one-field edit — the AutoVer CLI has no edit verb, and adding a third Major entry would have left the mislabeled Minor in place.

… fix branch name-drift message Address Copilot review on #2553: - CreateParallel 'name' XML docs said a name change 'does not break replay', but the name is passed to ValidateReplayConsistency (throws on drift). Doc now states the name is part of the deterministic definition and must stay stable. - Branch name-drift NonDeterministicExecutionException message had expected/found inverted; now reports the checkpointed name as expected and the current registration as the drifted value.

…al-parallel overflow/await fixes, serializer deferral, Branch rename, docs DurableExecution PR #2553 (stacked on feature/per-step-serializer-conformance). 1. [BLOCKER] StepOperation.ExecuteFunc: move the fresh-success SUCCEED enqueue and result round-trip OUTSIDE the try that funnels into HandleStepFailureAsync (mirrors ChildContextOperation). A serializer that cannot deserialize its own just-written payload now surfaces the fault directly instead of enqueuing a RETRY/FAIL that conflicts with the already-committed SUCCEED. 2. [MAJOR] .autover/changes/12a4a1f7: Minor -> Major. The package is GA (1.x) per CLAUDE.md, and the unconditional fresh-success round-trip is an observable happy-path behavior change for ALL serializers on non-suspending workflows (reference identity, DateTime.Kind, [JsonIgnore], precision). Not preview-exempt. 3. [MAJOR] IncrementalParallelOperation overflow recovery: isolate overflow-recovery re-runs (frozenStatus set) from _shortCircuitCts/_dispatchCts so a completion-policy short-circuit can no longer cancel them; and exclude frozen branches from the cooperative-bail arm so _result honors _frozenStatus (never resolves a frozen Succeeded branch to SkippedError, which made `await branch` throw while Status==Succeeded and lost the recovered value). 4. [MINOR] DurableContext.CreateParallel: defer LambdaSerializerHelper.GetRequired via a lazy factory (memoized in the operation). A workflow overriding the serializer on every branch no longer requires a global serializer at CreateParallel time (AOT / per-branch scenario). GetRequired is resolved only when a branch falls back. 5. [MINOR] Rename IDurableParallel.BranchAsync<T> -> Branch<T>. The method returns a handle synchronously (not a Task), so the Async suffix was misleading. Safe: the API is new/unreleased (absent on master). Updated the interface, impl, all call sites (conformance + tests), docs (parallel.md), and the AutoVer changelog text. 6. [MINOR] IDurableParallel.Branch XML doc: document ArgumentNullException, ObjectDisposedException, and NonDeterministicExecutionException in addition to InvalidOperationException. 7. [MINOR] IncrementalParallelBranch.ExecuteAsync: fault _result before rethrowing a workflow-level DurableExecutionException, so a caller that catches the fault out of CompleteAsync and then awaits the handle observes the fault instead of hanging. 8. [MINOR] Correct the IncrementalParallelOperation class summary and IParallelBranch.Index doc to reflect the 1-based operation-ID suffix (hash("{parentId}-{index+1}")). 9. [NIT] IncrementalParallelHeterogeneousTest: replace the tautological Contains("200") (satisfied by the "USD:4200" POCO branch) with the distinguishing token "Payment":200. Tests: added 4 unit tests (fresh-success round-trip deserialize failure surfaces without RETRY/FAIL; CreateParallel with no global serializer + per-branch overrides does not throw; deferred fallback still throws on a non-overriding branch; a branch faulting with a workflow-level error faults the handle instead of hanging). Build and Amazon.Lambda.DurableExecution.Tests pass (447/447, net10.0). Integration-test and deployed-function projects compile; the heterogeneous integration test requires an AWS deployment and was not run here. AutoVer: source changes are refinements to the two features already covered by the existing change files, so both existing entries were updated (12a4a1f7 -> Major; add-incremental changelog text updated for the Branch rename) rather than adding a new change file. Reclassifying 12a4a1f7's Type was a one-field edit — the AutoVer CLI has no edit verb, and adding a third Major entry would have left the mislabeled Minor in place.

… fix branch name-drift message Address Copilot review on #2553: - CreateParallel 'name' XML docs said a name change 'does not break replay', but the name is passed to ValidateReplayConsistency (throws on drift). Doc now states the name is part of the deterministic definition and must stay stable. - Branch name-drift NonDeterministicExecutionException message had expected/found inverted; now reports the checkpointed name as expected and the current registration as the drifted value.

…al-parallel overflow/await fixes, serializer deferral, Branch rename, docs DurableExecution PR #2553 (stacked on feature/per-step-serializer-conformance). 1. [BLOCKER] StepOperation.ExecuteFunc: move the fresh-success SUCCEED enqueue and result round-trip OUTSIDE the try that funnels into HandleStepFailureAsync (mirrors ChildContextOperation). A serializer that cannot deserialize its own just-written payload now surfaces the fault directly instead of enqueuing a RETRY/FAIL that conflicts with the already-committed SUCCEED. 2. [MAJOR] .autover/changes/12a4a1f7: Minor -> Major. The package is GA (1.x) per CLAUDE.md, and the unconditional fresh-success round-trip is an observable happy-path behavior change for ALL serializers on non-suspending workflows (reference identity, DateTime.Kind, [JsonIgnore], precision). Not preview-exempt. 3. [MAJOR] IncrementalParallelOperation overflow recovery: isolate overflow-recovery re-runs (frozenStatus set) from _shortCircuitCts/_dispatchCts so a completion-policy short-circuit can no longer cancel them; and exclude frozen branches from the cooperative-bail arm so _result honors _frozenStatus (never resolves a frozen Succeeded branch to SkippedError, which made `await branch` throw while Status==Succeeded and lost the recovered value). 4. [MINOR] DurableContext.CreateParallel: defer LambdaSerializerHelper.GetRequired via a lazy factory (memoized in the operation). A workflow overriding the serializer on every branch no longer requires a global serializer at CreateParallel time (AOT / per-branch scenario). GetRequired is resolved only when a branch falls back. 5. [MINOR] Rename IDurableParallel.BranchAsync<T> -> Branch<T>. The method returns a handle synchronously (not a Task), so the Async suffix was misleading. Safe: the API is new/unreleased (absent on master). Updated the interface, impl, all call sites (conformance + tests), docs (parallel.md), and the AutoVer changelog text. 6. [MINOR] IDurableParallel.Branch XML doc: document ArgumentNullException, ObjectDisposedException, and NonDeterministicExecutionException in addition to InvalidOperationException. 7. [MINOR] IncrementalParallelBranch.ExecuteAsync: fault _result before rethrowing a workflow-level DurableExecutionException, so a caller that catches the fault out of CompleteAsync and then awaits the handle observes the fault instead of hanging. 8. [MINOR] Correct the IncrementalParallelOperation class summary and IParallelBranch.Index doc to reflect the 1-based operation-ID suffix (hash("{parentId}-{index+1}")). 9. [NIT] IncrementalParallelHeterogeneousTest: replace the tautological Contains("200") (satisfied by the "USD:4200" POCO branch) with the distinguishing token "Payment":200. Tests: added 4 unit tests (fresh-success round-trip deserialize failure surfaces without RETRY/FAIL; CreateParallel with no global serializer + per-branch overrides does not throw; deferred fallback still throws on a non-overriding branch; a branch faulting with a workflow-level error faults the handle instead of hanging). Build and Amazon.Lambda.DurableExecution.Tests pass (447/447, net10.0). Integration-test and deployed-function projects compile; the heterogeneous integration test requires an AWS deployment and was not run here. AutoVer: source changes are refinements to the two features already covered by the existing change files, so both existing entries were updated (12a4a1f7 -> Major; add-incremental changelog text updated for the Branch rename) rather than adding a new change file. Reclassifying 12a4a1f7's Type was a one-field edit — the AutoVer CLI has no edit verb, and adding a third Major entry would have left the mislabeled Minor in place.

… fix branch name-drift message Address Copilot review on #2553: - CreateParallel 'name' XML docs said a name change 'does not break replay', but the name is passed to ValidateReplayConsistency (throws on drift). Doc now states the name is part of the deterministic definition and must stay stable. - Branch name-drift NonDeterministicExecutionException message had expected/found inverted; now reports the checkpointed name as expected and the current registration as the drifted value.

Adds an additive, branch-oriented parallel API alongside the existing homogeneous ParallelAsync<T> overloads: await using var parallel = ctx.CreateParallel(name: "process-order"); IParallelBranch<InventoryReservation> inv = parallel.BranchAsync("inventory", ...); IParallelBranch<PaymentAuthorization> pay = parallel.BranchAsync("payment", ...); IBatchResult summary = await parallel.CompleteAsync(); InventoryReservation r = await inv; // own type, no shared base/cast/envelope Each branch declares its own result type (heterogeneous) and returns an awaitable typed handle; branches are registered incrementally and start executing on registration (gated by MaxConcurrency); CompleteAsync seals, awaits per CompletionConfig, and checkpoints the aggregate. Implementation reuses the existing machinery so replay is identical to the batch API: each branch runs as a ChildContextOperation<T> with the same deterministic child op id (hash("{parentId}-{index}")) and the same parent CONTEXT/Parallel BatchSummary checkpoint shape. Terminal-parent replay reconstructs branch outcomes from the frozen inline summary (re-running only overflow-stripped branches); DisposeAsync auto-completes so `await using` always writes the terminal checkpoint. MaxConcurrency, CompletionConfig, NestingType, cancellation, and ILambdaSerializer are honored unchanged. Also factors the BatchSummary (de)serialization + overflow handling out of ConcurrentOperation<T> into a shared BatchSummaryCodec so the batch and incremental parallel paths cannot diverge on the wire format. New public API: - IDurableContext.CreateParallel(name?, config?) - IDurableParallel (BranchAsync<T>, CompleteAsync, IAsyncDisposable) - IParallelBranch<T> (Name/Index/Status, awaitable) Tests: 16 unit tests (IncrementalParallelOperationTests) covering fresh happy path, heterogeneous types, deterministic ids, MaxConcurrency, completion short-circuit/skip, failure surfacing, empty, replay reconstruct, name-drift, and STARTED-parent replay. Two integration tests (heterogeneous end-to-end and replay determinism across the Run and Terminal-reconstruct paths), both verified green against the durable execution service. All 428 unit tests pass; docs/core/parallel.md documents the new API.

…#2519)

…#2519) - Replay: switch explicitly on parent status — only SUCCEEDED reconstructs and only STARTED/PENDING re-run; any other terminal status (FAILED/CANCELLED/ STOPPED/TIMED_OUT) throws NonDeterministicExecutionException instead of silently re-running and overwriting the prior outcome (mirrors ConcurrentOperation.ReplayAsync). - CompleteAsync idempotence: cache the in-progress completion Task, not just the finished result, so concurrent CompleteAsync/DisposeAsync calls share one completion and enqueue exactly one parent SUCCEED. - Terminal replay: enforce the positional replay contract — the registered branch count must equal the frozen summary's unit count, else throw. - Percentage failure tolerance is no longer evaluated against the incomplete denominator during incremental registration; it is suppressed until the operation is sealed (CompletionPolicy gains an evaluatePercentage flag, defaulting true so batch behavior is unchanged). - Observe the per-branch result-task fault in the handle ctor so a discarded failed handle cannot surface as an UnobservedTaskException. - DisposeAsync no longer throws: its safety-net completion swallows faults. - Docs: correct the CreateParallel `name` param (positional op id, not name-derived); document that the CompleteAsync token governs sealing/awaiting and does not retroactively cancel already-started branch bodies. Adds 3 unit tests (unexpected-status throw, branch-count-mismatch throw, percentage-not-evaluated-before-seal). 431 unit tests pass; both incremental integration tests re-verified green against the durable execution service.

…r CreateParallel (#2519) Stacks on the per-step-serializer work: CreateParallel now honors ParallelConfig.ItemSerializer as the operation-level branch-result serializer, and IDurableParallel.BranchAsync accepts an optional per-branch ILambdaSerializer override (falls back to ItemSerializer, then the globally-registered serializer). Each branch's serializer is threaded into both its ChildContextOperation and the inline summary serialization so fresh and replay values match. Adds unit tests for per-branch and operation-level ItemSerializer, and relaxes the timing- sensitive FirstSuccessful test to its deterministic invariants.

* test(DurableExecution): add incremental parallel conformance handlers * test(DurableExecution): add dedicated static typing suite --------- Co-authored-by: Frank Chen <frankchn@dev-dsk-frankchn-2a-ff9871a5.us-west-2.amazon.com>

…al-parallel overflow/await fixes, serializer deferral, Branch rename, docs DurableExecution PR #2553 (stacked on feature/per-step-serializer-conformance). 1. [BLOCKER] StepOperation.ExecuteFunc: move the fresh-success SUCCEED enqueue and result round-trip OUTSIDE the try that funnels into HandleStepFailureAsync (mirrors ChildContextOperation). A serializer that cannot deserialize its own just-written payload now surfaces the fault directly instead of enqueuing a RETRY/FAIL that conflicts with the already-committed SUCCEED. 2. [MAJOR] .autover/changes/12a4a1f7: Minor -> Major. The package is GA (1.x) per CLAUDE.md, and the unconditional fresh-success round-trip is an observable happy-path behavior change for ALL serializers on non-suspending workflows (reference identity, DateTime.Kind, [JsonIgnore], precision). Not preview-exempt. 3. [MAJOR] IncrementalParallelOperation overflow recovery: isolate overflow-recovery re-runs (frozenStatus set) from _shortCircuitCts/_dispatchCts so a completion-policy short-circuit can no longer cancel them; and exclude frozen branches from the cooperative-bail arm so _result honors _frozenStatus (never resolves a frozen Succeeded branch to SkippedError, which made `await branch` throw while Status==Succeeded and lost the recovered value). 4. [MINOR] DurableContext.CreateParallel: defer LambdaSerializerHelper.GetRequired via a lazy factory (memoized in the operation). A workflow overriding the serializer on every branch no longer requires a global serializer at CreateParallel time (AOT / per-branch scenario). GetRequired is resolved only when a branch falls back. 5. [MINOR] Rename IDurableParallel.BranchAsync<T> -> Branch<T>. The method returns a handle synchronously (not a Task), so the Async suffix was misleading. Safe: the API is new/unreleased (absent on master). Updated the interface, impl, all call sites (conformance + tests), docs (parallel.md), and the AutoVer changelog text. 6. [MINOR] IDurableParallel.Branch XML doc: document ArgumentNullException, ObjectDisposedException, and NonDeterministicExecutionException in addition to InvalidOperationException. 7. [MINOR] IncrementalParallelBranch.ExecuteAsync: fault _result before rethrowing a workflow-level DurableExecutionException, so a caller that catches the fault out of CompleteAsync and then awaits the handle observes the fault instead of hanging. 8. [MINOR] Correct the IncrementalParallelOperation class summary and IParallelBranch.Index doc to reflect the 1-based operation-ID suffix (hash("{parentId}-{index+1}")). 9. [NIT] IncrementalParallelHeterogeneousTest: replace the tautological Contains("200") (satisfied by the "USD:4200" POCO branch) with the distinguishing token "Payment":200. Tests: added 4 unit tests (fresh-success round-trip deserialize failure surfaces without RETRY/FAIL; CreateParallel with no global serializer + per-branch overrides does not throw; deferred fallback still throws on a non-overriding branch; a branch faulting with a workflow-level error faults the handle instead of hanging). Build and Amazon.Lambda.DurableExecution.Tests pass (447/447, net10.0). Integration-test and deployed-function projects compile; the heterogeneous integration test requires an AWS deployment and was not run here. AutoVer: source changes are refinements to the two features already covered by the existing change files, so both existing entries were updated (12a4a1f7 -> Major; add-incremental changelog text updated for the Branch rename) rather than adding a new change file. Reclassifying 12a4a1f7's Type was a one-field edit — the AutoVer CLI has no edit verb, and adding a third Major entry would have left the mislabeled Minor in place.

…c; add overflow-recovery terminal-path tests

… fix branch name-drift message Address Copilot review on #2553: - CreateParallel 'name' XML docs said a name change 'does not break replay', but the name is passed to ValidateReplayConsistency (throws on drift). Doc now states the name is part of the deterministic definition and must stay stable. - Branch name-drift NonDeterministicExecutionException message had expected/found inverted; now reports the checkpointed name as expected and the current registration as the drifted value.

afroz0429 · 2026-09-23T19:14:36Z

With Claude's help:

PR #2553 review: blocking findings

ParallelConfig.ItemSerializer does not work on the new CreateParallel API. There are two separate causes, described below as 1a and 1b.

This combination is documented. docs/core/steps.md:196 states: "This works the same way for any per-operation serializer slot ... MapConfig<TItem>/ParallelConfig.ItemSerializer". It works on batch ParallelAsync today, so anyone moving from ParallelAsync to CreateParallel with a per-operation serializer hits this on the first invocation.

Reproduced on cd03b680 using the SDK's own FileSystemSerializer, with local xunit tests against RecordingBatcher. No AWS required. Same config on both APIs:

batch ParallelAsync  => success=2 hasFailure=False values=[alpha,beta]   files written=2
CreateParallel       => every branch fails (1a), or is recorded failed while
                        returning its value (1b)

Existing ParallelAsync<T> and MapAsync behavior is unaffected by this PR.

1a. CreateParallel does not bind the inner serializer

Location: DurableContext.cs, the CreateParallel block. The call is missing.

Every other serializer-resolving dispatcher in DurableContext passes the effective serializer through LambdaSerializerHelper.WithDefaultInner(...). There are 5 such call sites, including ParallelAsync at :276 and MapAsync at :310:

var serializer = LambdaSerializerHelper.WithDefaultInner(
    effectiveConfig.ItemSerializer ?? defaultSerializer, defaultSerializer);

CreateParallel skips this and passes effectiveConfig.ItemSerializer straight through, so a serializer implementing IDefaultInnerSerializer never receives its inner. With the documented inner-less FileSystemSerializer constructor, every branch fails:

inner-less FileSystemSerializer in ParallelConfig.ItemSerializer:
  batch ParallelAsync => success=2 hasFailure=False   <- works
  CreateParallel      => a.Status=Failed success=0 failure=1
  error => FileSystemSerializer was constructed without an inner serializer and the
           durable runtime did not supply a globally-registered ILambdaSerializer to
           use as the inner.

Customer impact: the documented inner-less pattern does not work on the new API. The failure is immediate and visible, so nothing is silently corrupted, but the feature is unusable.

Fix: add the WithDefaultInner call to match the other five dispatchers.

1b. A branch's recorded outcome can contradict its observed result

Location: Internal/IncrementalParallelOperation.cs:166-169 and :250-263.

In IncrementalParallelBranch<T>.ExecuteAsync, the public result task completes before the value is serialized:

var value = await run().ConfigureAwait(false);
if (_frozenStatus is null) _status = (int)BatchItemStatus.Succeeded;
_result.TrySetResult(value);                                   // completes the handle
return BranchOutcome.Success(Index, Name, Serialize(value));   // can throw

If Serialize throws, catch (Exception) sets _status = Failed and calls TrySetException. That call does nothing, because _result has already completed successfully. The branch is then checkpointed as FAILED while its handle still yields the value.

Serialize at :250 calls _serializer.Serialize(value, ms) directly instead of LambdaSerializerHelper.Serialize(..., DurableSerializationContext). The branch's own child checkpoint is written correctly through the helper. The problem is the second, redundant re-serialization done for the parent's inline summary, which bypasses the helper and therefore hits the plain ILambdaSerializer path. Batch Parallel makes no such call. It reuses the child's already-written payload (nestedInlinePayloads).

With an explicit-inner FileSystemSerializer, so that 1a is out of the way and the child checkpoint succeeds, both branches invert:

await a / await b   => "alpha" / "beta"          <- values returned
a.Status / b.Status => Failed / Failed
summary             => success=0 failure=2 hasFailure=True reason=FailureToleranceExceeded
checkpoint          => {"CompletionReason":"FAILURE_TOLERANCE_EXCEEDED","Units":[
                        {"Index":0,"Name":"a","Status":"FAILED","Result":null,
                         "Error":{"ErrorType":"System.NotSupportedException", ...}}, ...]}

Customer impact: the live invocation reads "alpha" and takes the success path. The durable record says the branch failed, so on replay await a throws and the workflow takes the failure path. A workflow can ship an order on the first invocation and refund it on the replay.

Step_FreshSuccess_RoundTripDeserializeFailure_FailsTerminallyWithoutRetry, added in this PR, locks in the opposite behavior for StepAsync: a serialization failure after the body has succeeded must fail terminally before SUCCEED is written, rather than diverge silently.

Fix: serialize before TrySetResult. Route the call through LambdaSerializerHelper with new DurableSerializationContext(childOpId, _durableExecutionArn), or reuse the child's payload the way batch Parallel does. Treat a serialize failure as a terminal branch failure.

…llel Address PR review findings on the incremental parallel API: - Bind the global inner serializer in CreateParallel via LambdaSerializerHelper.WithDefaultInner, matching the other dispatchers, so an inner-less ParallelConfig.ItemSerializer (e.g. FileSystemSerializer) works as it does on batch ParallelAsync (1a). - Produce each branch's inline summary payload through LambdaSerializerHelper (reusing the Nested child's own payload, or serializing a Flat branch with the correct entity id) BEFORE completing the branch handle, so a serialize failure is a terminal branch failure and the recorded outcome can no longer contradict the observed handle result. Terminal-reconstruct deserialize is routed through the context-aware path too (1b). - Recover a terminal parent with a missing/corrupt summary payload from the surviving per-branch child checkpoints instead of masking every branch as skipped and synthesizing a false AllCompleted. Adds 4 regression tests.

afroz429 · 2026-09-23T19:40:37Z

Garrett resolved the above findings in commit 4134e2b

GarrettBeatty requested a balanced review from Copilot September 1, 2026 22:34

Copilot started reviewing on behalf of GarrettBeatty September 1, 2026 22:45 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

This was referenced Sep 2, 2026

test(DurableExecution): add static-typing parallel suite #2554

Merged

feat: add optional static-typing conformance suite aws/aws-durable-execution-conformance-tests#112

Merged

GarrettBeatty changed the base branch from master to feature/per-step-serializer-conformance September 2, 2026 21:09

GarrettBeatty force-pushed the feature/heterogeneous-parallel branch from 43ff3ef to 3fd7096 Compare September 2, 2026 21:10

GarrettBeatty requested a balanced review from Copilot September 2, 2026 21:11

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread Libraries/src/Amazon.Lambda.DurableExecution/Internal/IncrementalParallelOperation.cs

Comment thread Libraries/src/Amazon.Lambda.DurableExecution/Internal/IncrementalParallelOperation.cs Outdated

Copilot started reviewing on behalf of GarrettBeatty September 2, 2026 21:29 View session

GarrettBeatty force-pushed the feature/per-step-serializer-conformance branch 2 times, most recently from cfac883 to 545df80 Compare September 3, 2026 03:23

GarrettBeatty force-pushed the feature/heterogeneous-parallel branch from 3fd7096 to 9b1220e Compare September 3, 2026 03:35

GarrettBeatty force-pushed the feature/heterogeneous-parallel branch from 9b727b7 to c760c01 Compare September 3, 2026 03:51

GarrettBeatty changed the base branch from feature/per-step-serializer-conformance to feature/per-step-serializer September 3, 2026 03:51

GarrettBeatty force-pushed the feature/heterogeneous-parallel branch from 51d7258 to b244c21 Compare September 3, 2026 18:02

GarrettBeatty changed the base branch from feature/per-step-serializer to feature/durable-result-serializer September 4, 2026 01:05

GarrettBeatty force-pushed the feature/heterogeneous-parallel branch from 147d7a4 to 075c4d8 Compare September 4, 2026 01:16

GarrettBeatty force-pushed the feature/heterogeneous-parallel branch from 075c4d8 to 8019228 Compare September 4, 2026 01:17

GarrettBeatty changed the base branch from feature/durable-result-serializer to master September 9, 2026 13:56

GarrettBeatty added 2 commits September 11, 2026 18:50

chore: add AutoVer change file for incremental heterogeneous Parallel (…

5fae3c2

…#2519)

GarrettBeatty and others added 6 commits September 11, 2026 18:50

Address adversarial review round 2: fix CompleteAsync cancellation do…

863a996

…c; add overflow-recovery terminal-path tests

GarrettBeatty force-pushed the feature/heterogeneous-parallel branch from 8019228 to cd03b68 Compare September 11, 2026 22:51

GarrettBeatty marked this pull request as ready for review September 14, 2026 14:50

GarrettBeatty requested review from a team as code owners September 14, 2026 14:50

GarrettBeatty requested review from afroz429 and normj September 14, 2026 14:50

normj reviewed Sep 14, 2026

View reviewed changes

Comment thread Libraries/src/Amazon.Lambda.DurableExecution/Internal/CompletionPolicy.cs

GarrettBeatty requested a review from normj September 14, 2026 17:40

normj approved these changes Sep 16, 2026

View reviewed changes

afroz429 approved these changes Sep 23, 2026

View reviewed changes

afroz429 merged commit 6ca28fc into master Sep 23, 2026
19 checks passed

afroz429 deleted the feature/heterogeneous-parallel branch September 23, 2026 20:54

Navigation Menu

feat(DurableExecution): incremental, heterogeneous Parallel API (#2519) - #2553

feat(DurableExecution): incremental, heterogeneous Parallel API (#2519)#2553
afroz429 merged 9 commits into
masterfrom
feature/heterogeneous-parallel

GarrettBeatty commented Sep 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

afroz0429 commented Sep 23, 2026

Uh oh!

afroz429 commented Sep 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

GarrettBeatty commented Sep 1, 2026

Description

What & why

New public API

Design

Testing

Note for reviewers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (2)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

afroz0429 commented Sep 23, 2026

PR #2553 review: blocking findings

1a. CreateParallel does not bind the inner serializer

1b. A branch's recorded outcome can contradict its observed result

Uh oh!

afroz429 commented Sep 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants